Entropy Message Passing
نویسندگان
چکیده
The paper proposes a new message passing algorithm for cycle-free factor graphs. The proposed ”entropy message passing” (EMP) algorithm may be viewed as sumproduct message passing over the entropy semiring, which has previously appeared in automata theory. The primary use of EMP is to compute the entropy of a model. However, EMP can also be used to compute expressions that appear in expectation maximization and in gradient descent algorithms.
منابع مشابه
Entropy Message Passing Algorithm
Message passing over factor graph can be considered as generalization of many well known algorithms for efficient marginalization of multivariate function. A specific instance of the algorithm is obtained by choosing an appropriate commutative semiring for the range of the function to be marginalized. Some examples are Viterbi algorithm, obtained on max-product semiring and forward-backward alg...
متن کاملUnsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
Unsupervised neural network learning extracts hidden features from unlabeled training data. This is used as a pretraining step for further supervised learning in deep networks. Hence, understanding unsupervised learning is of fundamental importance. Here, we study the unsupervised learning from a finite number of data, based on the restricted Boltzmann machine where only one hidden neuron is co...
متن کاملThe Entropy Message Passing: A New Algorithm Over Factor Graphs
Message passing over a factor graph can be considered as a generalization of the many well-known algorithms for an efficient marginalization of multivariate functions. The specific instance of the algorithm is obtained by selection of an appropriate commutative semiring for the range of the function to be marginalized. Some of the examples are the Viterbi algorithm, obtained on a max-product se...
متن کاملGradient computation in linear-chain conditional random fields using the entropy message passing algorithm
The paper proposes a new recursive algorithm for the exact computation of the linear chain conditional random fields gradient. The algorithm is an instance of the Entropy Message Passing (EMP), introduced in our previous work, and has the purpose to enhance memory efficiency when applied to long observation sequences. Unlike the traditional algorithm based on the forward and the backward recurs...
متن کاملSparse Estimation Based on a New Random Regularized Matching Pursuit Generalized Approximate Message Passing Algorithm
Yongjie Luo 1, Guan Gui 2,*, Xunchao Cong 1 and Qun Wan 1 1 Department of Electronic Engineering, University of Electronic Science and Technology of China, No. 2006, Xiyuan Ave., West Hi-Tech Zone, Chengdu 611731, China; [email protected] (X.C.); [email protected] (Q.W.) 2 College of Telecommunication and Information Engineering, Nanjing University of Posts and Telecommunications,...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- IEEE Trans. Information Theory
دوره 57 شماره
صفحات -
تاریخ انتشار 2011